data-shrinker


a very light and fast compression program with acceptable ratio

It's a LZ77-based data compression program that can be used in high performance demand environment, it's can remove much of the redundancy in data at the speed of hundreds of megabytes per second.


The encoding scheme is inspired by and very similar to LZ4. The modifications may lead to some compression improvement, especially on binary data.

The implement is very simple, only about 200 lines of code in total. The compression state data can be in stack. Just call one function at a (de)compression operation.

Currently, the underlying theory do not differ much among the state of art light weight compressors, such as quicklz, liblzf, snappy, lz4 etc.. The shrinker can be an alternative choice. Actually, it outperformed others on either size or speed or both in much of my test cases. I suggest to test them on your own data which you work on.

Benchmark is welcomed!


A quick benchmark on Pentium Dual-Core T4200 2.0GHz, 2G DDR2 667; Test tool fs-bench; Test corpus Maximumcompression-SFC

memcpy               = 111 ms (3195 MB/s), 53135166->53135166
Codec version args Size (Ratio) C.Speed D.Speed
fastlz 0.1.0 1 21588439 (x 2.46) C: 145 MB/s D: 328 MB/s
LZ4 r59 12 21044733 (x 2.52) C: 274 MB/s D: 734 MB/s
LZF 3.6 very 21247147 (x 2.50) C: 160 MB/s D: 368 MB/s
lzjb 2010 25167107 (x 2.11) C: 133 MB/s D: 269 MB/s
QuickLZ 1.5.1b6 1 20159761 (x 2.64) C: 239 MB/s D: 225 MB/s
Shrinker r5 19342542 (x 2.75) C: 169 MB/s D: 487 MB/s
Snappy 1.0.5 21863187 (x 2.43) C: 208 MB/s D: 518 MB/s
zlib 1.2.5 6 15095953 (x 3.52) C: 14 MB/s D: 183 MB/s

Project Information

The project was created on Mar 18, 2012.

Labels:
compression fast light C lz77